ABSTRACT The goal of this project is to plan the development of a richly annotated repository of 3D models called ShapeNet that currently exists only in a preliminary form. ShapeNet will include 3-4 million 3D models of everyday objects in 4-5 thousand categories, in a variety of representations. Models in the ShapeNet repository will be annotated with multiple annotation types: geometric (parts, symmetries), semantic (keywords for the shape and its parts), physical (weight, size), and functional (affordances, scene context). The availability of ShapeNet data, capturing the 3D geometry of a significant fraction of object categories in the world, together with associated detailed meta-data and semantic information, will catalyze major developments in graphics, vision and robotics by providing adequate data against which new proposed techniques and methodologies for shape or scene analysis and synthesis can be vetted -- and with which machine learning algorithms can be trained. ShapeNet can be considered an encyclopedia that facilitates the creation of intelligent systems and agents capable of operating autonomously in the world --- because they can have deep knowledge of that world.

While most of the ShapeNet models will be initially found on the Web, the annotations will be obtained through an active learning combination of modest human input (including crowd-sourcing), extensive algorithmic transport, and human verification. During the planning period the effort will focus on mathematical representations of the semantic knowledge associated with 3D models, as well as on a design framework for key algorithms allowing knowledge transport from one model to another. Further challenges to be addressed include the quantification of data quality issues and the specification of all the multimodal (3D, image, language) UIs and APIs needed for users to be able to exploit and search this wealth of data, or to contribute additional models and annotations to it.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval). Some links on this page may take you to non-federal websites. Their policies may differ from this site.

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

This project has studied how to leverage relationships between multiple correlated data sets so as to improve our understanding of each data set separately. Our primary focus has been information propagation among visual data sets, such as 2D images and videos, or 3D models and scans. We have explored how to start with noisy maps and correspondences connecting multiple such data sets and clean them by exploiting consistency conditions among the maps, even in the presence of ambiguities caused by symmetry. In supervised learning tasks, we have shown how human part annotations on a sparse set of 3D models can be propagated to many more related models, easing the annotation burden. We have also demonstrated texture transfer from object images to 3D models, enriching the visual fidelity of the latter. In general, our tools facilitate the reliable transport of information between data sets, thus enabling novel collaboration methodologies between communities of people (e.g., scientists, doctors, etc.) with shared data of interest (e.g., images, 3D scans, graphs, etc.). The project has produced multiple publications in top venues in computer vision, computer graphics, and machine learning. It also contributed to data sets useful to the research community -- one example is the largest extant repository of 3D models with detailed semantic part annotations (ShapeNetCore).

Last Modified: 12/15/2018 Modified by: Leonidas J Guibas

Please report errors in award information by writing to: [email protected].